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VINCENT VATTER 

Abstract. Zeilberger's enumeration schemes can be used to completely automate the enu- 
meration of many permutation classes. We extend his enumeration schemes so that they 
apply to many more permutation classes and describe the Maple package WilfPlus, which 
implements this process. We also compare enumeration schemes to three other systematic 
enumeration techniques: generating trees, substitution decompositions, and the insertion en- 
coding. 



1. Introduction 

The enumeration of permutation classes, whose ancestry can be traced back to at least 
1915 (MacMahon [34]), has frequently been accomplished by beautiful arguments utilizing 
such diverse objects as Young tableaux, Dyck paths, and planar maps, to name only a few. 
Our concern herein is not with attractive proofs, but rather with systematic methods for 
solving the enumeration problem. We adopt a strict definition of systematic, insisting that the 
computations can be performed without any human interaction whatsoever. For the definition 
of enumeration, we follow Wilf [57] and insist only on a polynomial time (in n) algorithm to 
compute the number of length n permutations in the class. We refer to such an algorithm as a 
Wilfian formula. To date, four techniques with wide applicability have been introduced which 
satisfy these goals: 

• generating trees, 

• enumeration schemes, 

• substitution decompositions, 

• the insertion encoding. 

The major aim of this paper, carried out in Section 6, is to extend the method of enumeration 
schemes so that it can enumerate a wider variety of permutation classes and describe the Maple 
package WilfPlus, which can rigorously and automatically find these extended schemes. 
Before that, we briefly examine the other methods in Sections 2-4 and review enumeration 
schemes in Section 5. Section 7 contains examples of classes which lie beyond the reach of even 
our more powerful enumeration schemes, while Section 8 gives numerous examples which can 
be handled. First we describe permutation classes. 

Two sequences of natural numbers are said to be order isomorphic if they have the same 
pairwise comparisons, so 9,1,6,7,2 is order isomorphic to 5,1,3,4,2. Every sequence w of 
natural numbers without repetition is order isomorphic to a unique permutation that we denote 
by st(w), so st(9, 1, 6, 7, 2) = 5, 1, 3, 4, 2, which we shorten to 51342. We say that st(w) is the 
standardization of w. We further say that the permutation tt contains the permutation (3 if 
7T contains a subsequence that is order isomorphic to 0, and in this case we write (3 < ir. For 
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Figure 1. A depiction of the applicability of the four systematic enumeration techniques 



example, 391867452 contains 51342, as can be seen by considering the subsequence 91672. A 
permutation is said to avoid another if it does not contain it. 

A permutation class is a lower order ideal in the containment ordering, meaning that if ir 
is contained in a permutation in the class, then ir itself lies in the class. Permutation classes 
can be specified in terms of the minimal permutations not lying in the class, which we call 
the basis of the class. By this minimality condition, bases are necessarily antichains, meaning 
that no element of a basis is contained in another. Although there are infinite antichains of 
permutations (see Atkinson, Murphy, and Ruskuc [7] for constructions and references to earlier 
work), we restrict our attention to finitely based classes. Given a set of permutations B, we 
define Av(-B) to be the set of permutations that avoid all of the permutations in B. Thus 
if C is a closed class with basis B then C = Av(B), and for this reason the elements of a 
permutation class are often referred to as restricted permutations. We let s n {B) denote the 
number of permutations of length n in Av(-B), and refer to ^2 n s n (B)x n as the generating 
function of Av(B). For more information on permutation classes, the reader is referred to 
Bona's text [11]. 

Each of the four systematic approaches for permutation class enumeration has a natural 
notion of a "state," and in each case if the class is such that only finitely states are needed 
then — at least in principle — these methods give a Wilfian formula for the number of length 
n permutations in the class. For generating trees, the states are the labels of the isomorphic 
generating tree. The classes possessing a generating tree with only finitely many labels are 
characterized in Vatter [54]; this characterization appears here as Theorem 2.1. For the in- 
sertion encoding, which associates a language to the permutation class, the natural notion of 
"state" is a state in the accepting automaton for the associated language. The classes that 
require only finitely many states (or in other words, the classes that correspond to regular 
languages) were characterized by Albert, Linton, and Ruskuc [3]; their result appears here as 
Theorem 3.1. For enumeration schemes the translation of "state" is "ES + -irreducible permu- 
tation" (or, for Zeilberger's original schemes, "ES-irreducible permutation"). Should a class 
contain only finitely many such permutations then WilfPlus can automatically enumerate 
it. No characterization of these classes is known 1 . Moreover, unlike the other methods, there 
are subclasses of classes with finite enumeration schemes which do not themselves have finite 
enumeration schemes 2 , indicating that such a characterization may be too much to hope for. 
For substitution decompositions, simple permutations play the role of states. As with enu- 
meration schemes, there is no known characterization of the classes that contain only finitely 
many simple permutations. 



Zeilberger [59] dismisses this by stating "if we know beforehand that we are guaranteed to succeed, then it 
is not research, but doing chores." 

2 In fact, the set of all permutations, Av(0), has a finite enumeration scheme (shown in Figure 4 on page 14), 
while several examples of classes without finite enumeration schemes are given in Section 7. 



ENUMERATION SCHEMES FOR RESTRICTED PERMUTATIONS 



3 



The classes that these techniques can automatically enumerate are related as shown in 
Figure 1, which is to say, they are not very closely related at all (this is established via a 
series of examples in Section 7 and remarks in Sections 2 and 3) . Care should be taken when 
reading one symbol in this diagram; while the inclusion from finitely labeled generating trees to 
finite enumeration schemes indicates an increase in the number of classes that can be counted, 
there is a corresponding decrease in information. Finitely labeled generating trees and regular 
insertion encodings show that a class has a rational generating function, while classes with only 
finitely many simple permutations have algebraic generating functions. It is not yet known 
what types of generating functions can arise from finite enumeration schemes, but they need 
not be algebraic. For example, Av(1234), which has a holonomic 3 but non-algebraic generating 
function (see Gessel [24]), has a finite enumeration scheme 4 (shown in Figure 6 on page 15). 
It is natural to hope that finite enumeration schemes produce only holonomic sequences, but 
this hope remains unproven. 

Perhaps the greatest loss of information occurs with Wilf-equivalence. Two classes are said 
to be Wilf-equivalent if they are equinumerous. Clearly taking the reverse of a class yields 
a Wilf-equivalent class, as does taking the inverse, and these two operations generate the 
dihedral group with eight elements 5 . However, many examples of non-trivial Wilf-equivalences 
have been observed, ranging from the fact every class defined by avoiding a single pattern 
of length three is Wilf-equivalent 6 to the theorem of Atkinson, Murphy, and Ruskuc [6] that 
Av(1342) is Wilf-equivalent to the infinitely based class 

Av({2(2m - 1)416385 • • • (2m)(2m - 3) : m = 2, 3, . . . }). 

If two classes both have finitely labeled generating trees, regular insertion encodings, or 
finitely many simple permutations then, since we can compute their generating functions from 
this information, we can decide whether or not they are Wilf-equivalent. For enumeration 
schemes this issue is not so clear. Occasionally, as with the enumeration schemes pictured in 
Figure 9 (a) and (b) on page 17, the Wilf-equivalence of two classes can be easily deduced from 
their enumeration schemes, but we present several examples in Section 8 where such deductions 
do not readily present themselves. 

2. Generating trees 

Generating trees were introduced by Chung, Graham, Hoggatt, and Kleiman [16] and became 
quite popular after a pair of articles by West [55, 56]. The closely related ECO (enumerating 
combinatorial objects) method (see Barcucci, Del Lungo, Pergola, and Pinzani [9] for a survey) 
extends the notion of generating trees to other settings. 

We say that the permutation a of length n is a child of ir £ S n -i if a can be obtained by 
inserting n into it. This defines a rooted tree T on the set of all permutations. The pattern- 
avoidance tree of Av(fi), denoted by T(B), is then the subtree of T with nodes Av(B). For 
example, the first four levels of T(132, 231) are shown in Figure 2. 

generating function is said to be holonomic (or synonymously in the univariate case, D-finite) if its 
derivatives span a finite dimensional subspace over C(x). This is equivalent to the corresponding sequence 
s n being holonomic (again synonymously in the univariate case, P -recursive) , which means that there are 

polynomials po,Pi, ■■ -Pk so that p k {ri)s n+k + p k -i(n)s n+k -i + \-po(n)s„ = 0. 

4 A more trivial example would be the class of all permutations. 

5 With the exception of substitution decompositions, these techniques are not invariant under the eight 
permutation class symmetries. To be precise, there are classes that cannot be handled with these methods, 
while their inverses can be handled easily. Thus the comment of Albert, Linton, and Ruskuc [3] that "this 
apparent asymmetry does represent a possible flaw of the insertion encoding in general" applies equally well to 
enumeration schemes and generating trees. 

6 The classical bijective proof of this result is due to Simion and Schmidt [46]. Zeilberger [58] gives a proof 
using a technique quite like enumeration schemes that generalizes to permutations of a multiset. 
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Figure 2. The first four levels of the pattern-avoidance tree T(132, 231) 

A generating tree, on the other hand, is a rooted, labeled tree such that the labels of the 
children of each node are determined by the label of that node. Sometimes the labels of the 
tree are taken to be natural numbers, but this is not necessary and frequently inconvenient. 
One specifies a generating tree by supplying the label of the root (also sometimes called the 
axiom) and a set of succession rules (also referred to as inductive steps). For example, the 
complete binary tree may be given by 

Root: (2) 

Rule: (2) ~» (2) (2). 

In order to enumerate the permutation class Av(B), we want to find a generating tree 
isomorphic (as a rooted tree) to T(B). For example, consider T(132, 231). We may obtain 
a permutation in Av n (132,231) by inserting n either at the beginning or the end of any 
7r G Av n _i(132, 231), but nowhere in between, so T(132,231) is isomorphic to the complete 
binary tree and thus to the generating tree given above. For a more complicated example we 
turn to T(1234), first described by West [55]. This tree is isomorphic to generating tree defined 
by 

Root: (2, 2) 

Rule: (s,t) •w (2, t + 1)(3,< + 1) ■ ■ ■ (s, t + l)(s, s + l)(s, s + 2) • • • (s, i)(s + 1, t + t). 

While verifying this isomorphism is not difficult (consider the lexicographically first ascent and 
the lexicographically first occurrence of 123 in the permutation), it is much harder to obtain the 
generating function for Av(1234) from this tree; for the details of this see Bousquet-Melou [13]. 

Let T(B;tt) denote the subtree of T(B) that is rooted at tt and contains all descendants of 
7T. In an isomorphism between T(B) and a generating tree, every permutation of Av(B) is 
assigned a label. Clearly two permutations tt and a may be assigned the same label if and only 
if T(B;ir) and T(B;a) are isomorphic (again, as rooted trees). Thus each pattern-avoidance 
tree T(B) is isomorphic to a canonical generating tree whose labels correspond exactly to the 
isomorphism classes of {T(B;ir) : tt G Av (£?)}. 

In particular, T(B) is isomorphic to a finitely labeled generating tree if and only if the set of 
all principal subtrees {T(B;ir) : tt G Av(£>)} contains only finitely many isomorphism classes. 
When this occurs, Av(B) has a rational generating function which may be routinely computed 
using the transfer matrix method (see Stanley's text [51, Section 4.7] for details). The finitely 
based classes for which this is possible are characterized by the following theorem. 

Theorem 2.1 (Vatter [54]). LetC be a finitely based permutation class. The pattern- avoidance 
tree of C is isomorphic to a finitely labeled generating tree if and only if C omits both a child 
of an increasing permutation and a child of a decreasing permutation. 

For example, T(132, 231) satisfies the hypotheses of Theorem 2.1 because it omits both 132 
(a child of the increasing permutation 12) and 231 (a child of 21). A less trivial example is 
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given by T(123, 3214, 2143, 15432), which arose in Klazar [28]. The Maple package FinLa- 
BEL (described in [54] and available at http://math.rutgers.edu/~vatter/) can find the 
generating functions for classes satisfying Theorem 2.1 completely automatically. 

It is easy to see that the hypotheses of Theorem 2.1 are necessary 7 . The other direction 
is proved by showing that every sufficiently long permutation is "GT-reducible." Since GT- 
reducibility is a stronger condition than the ES-reducibility of enumeration schemes, every 
class with a finitely labeled generating tree has a finite enumeration scheme. 

3. The insertion encoding 

The insertion encoding, recently introduced by Albert, Linton, and Ruskuc [3], is a cor- 
respondence between permutation classes and languages. With it, one may attack the enu- 
meration problem with all the tools of formal language theory. Roughly, this correspondence 
associates to each permutation a word describing how that permutation evolved. At each stage 
until the desired permutation has been constructed, at least one open slot (represented by a 
o) exists in the intermediate configuration, and to proceed to the next configuration we insert 
a new maximal entry into one of these slots. This insertion can occur in four possible ways: 

• the slot can be filled (replacing a o by n), 

• the new entry can be inserted to the left of the slot (replacing a o by no), 

• the new entry can be inserted to the right (replacing a o by o n) , or 

• the slot can be divided into two slots with the new entry in between (replacing a by 
ono). 

These operations are denoted by the symbols f , 1, r, and m, respectively. Since each of these 
operations can be performed on any open slot at any stage, we subscript their symbols with the 
number of the slot they were applied to (read from left to right). For example, the permutation 
31254 has the insertion encoding mi^firifi because its evolution is 

o 
olo 
ol2o 
312o 
312 o 4 
31254 

Let SB{k) denote the permutation class whose basis consists of all length 2k+\ permutations 
of the form babab ■ ■ ■ bab where the a's represent the elements {1,2,... , k} and the 6's represent 
the elements {k + 1, k + 2, . . . , 2k + 1}. These classes are called slot bounded because in the 
evolution of a permutation in SB{k) there are never more than k open slots. 

Theorem 3.1 (Albert, Linton, and Ruskuc [3]). The insertion encoding of a finitely based 
class is regular if and only if the class is a subclass of SB(k) for some k. 

One can show using the Erdos-Szekeres theorem [22] (or one can refer to the proof in [3]) 
that Theorem 3.1 includes all of the classes identified by Theorem 2.1 as having finitely labeled 
generating trees. 

Even when the insertion encoding of a class is not regular, useful information can still be ob- 
tained by this correspondence. For example, Albert, Elder, Rechnitzer, Westcott, Zabrocki [1] 
used regular approximations to the insertion encoding of Av(1324) to establish that s n (1324) > 
9.35 n for sufficiently large n, thereby disproving a conjecture of Arratia [4]. Additionally, Al- 
bert, Linton, and Ruskuc [3] consider several classes with context-free insertion encodings and 



Suppose, without loss, that C contains all children of every increasing permutation. Then for each n, 12 ■ ■ ■ n 
has n + 1 children in the pattern-avoidance tree of C, and thus no two of these nodes may share the same label. 
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are able to obtain their (algebraic) generating functions from these languages. However, the 
derivation of insertion encodings is only automatic for subclasses of SB(k), and thus we choose 
to limit our focus to this case. 



Substitution decompositions (also known as modular decompositions, disjunctive decompo- 
sitions, and X-joins) have proven to be a useful technique in a wide range of settings, ranging 
from game theory to combinatorial optimization (see Mohring [40] or Mohring and Raderma- 
cher [41] for extensive references). Permutation class enumeration is no exception. 

An interval (also called a block, or in other contexts, factor, clan, or even convex subset) in 
the permutation tt is an interval of indices I = [a, b] such that the set of values {7r(i) : i £ 1} 
also forms an interval. Clearly every permutation of length n has n trivial intervals of length 
one and one trivial interval of length n. A permutation that has no non-trivial intervals is 
called simple (the analogous term in other contexts is often prime or primitive). 

Simple permutations first appear in the work of Atkinson and Stitt [8], which is followed 
up by Albert and Atkinson [2]. Although in other contexts substitution decompositions are 
most often applied to algorithmic problems, they also have powerful enumerative applications. 
A class with only finitely many simple permutations has a recursive structure in which long 
permutations are built up from smaller permutations (their intervals). Thus it is natural to 
expect these classes to have algebraic generating functions, and this intuition is borne out by 
the following theorem. 

Theorem 4.1 (Albert and Atkinson [2]). A permutation class with only finitely many simple 
permutations has an algebraic generating function. 

The canonical example of a class with only finitely many simple permutations is Av(132). 
By considering the entries to the left and to the right of the n in a permutation in Av n (132) 
one simultaneously derives a decomposition of these permutations that leads immediately to 
the Catalan numbers and sees that this class contains no simple permutations of length three 8 
or longer. 

Another example of a class with only finitely many simple permutations is the class of sepa- 
rable permutations. This class, first introduced by Bose, Buss, and Lubiw [12], is essentially the 
permutation analogue of series-parallel posets (see Stanley [50, 51]) and complement reducible 
graphs (see Corneil, Lerchs, and Burlingham [17]). To define separable permutations we first 
need two binary operations on permutations. Given two permutations tt £ S m and a £ S n we 
define their direct sum, written tt © a, by 



Given a class C, we denote by sc(C) the strong completion of C, which is the smallest class 
containing C such that both tt © a and tt © o lie in sc(C) for every tt, a G sc(C). 

The separable permutations are the strong completion of {1}. As was shown by Bose, Buss, 
and Lubiw [12], this class can also be described as Av(2413, 3142). The enumeration of this 
class (which can now be seen to follow routinely from Theorem 4.1 and the fact that the only 
simple separable permutations are 1, 12, and 21) was first undertaken by West [55]. He used 
generating trees to show that the separable permutations are counted by the large Schroder 

ctually, there are no simple permutations of length three, 132-avoiding or otherwise. 



4. Substitution decompositions 




Similarly, we define their skew sum, tt © a, by 




ENUMERATION SCHEMES FOR RESTRICTED PERMUTATIONS 



7 



numbers. Later, Ehrenfeucht, Harju, ten Pas, and Rozenberg [19] (who also gave another 
proof that the basis of this class is {2413,3142}) presented a bijection between separable 
permutations and parenthesis words, the objects Schroder was originally interested in counting. 

One of the notable features of Theorem 4.1 is that it does not seem to require the class to 
be finitely based. However, this is merely an illusion: 

Theorem 4.2 (Albert and Atkinson [2], Murphy [42]). A permutation class with only finitely 
many simple permutations is both finitely based and partially well-ordered 9 . 

There is a semi-algorithm for establishing that a class contains only finitely many simple 
permutations. This semi-algorithm stems from the following theorem of Schmerl and Trot- 
ter [45], who proved it in the more general context of binary relational systems. Versions of 
the theorem for 2-structures and fc-structures are given by Ehrenfeucht and Rozenberg [21] and 
Ehrenfeucht and McConnell [20] , repectively, and a proof for the special case of permutations 
can be found in Murphy's thesis [42]. 

Theorem 4.3 (Schmerl and Trotter [45]). Every simple permutation of length n > 2 contains 
a simple permutation of length n — 1 or n — 2. 

If the class C contains only finitely many simple permutations, then clearly there is an integer 
n so that C does not contain any simple permutations of lengths n — 1 or n — 2. In the other 
direction, Theorem 4.3 shows that if we have found such an integer n then C contains no simple 
permutations of length n — 2 or longer. Therefore, when a class happens to contain only finitely 
many simple permutations, this fact can be verified automatically. 

It remains an interesting open question if it is decidable whether a class contains only finitely 
many simple permutations. 

5. ZEILBERGER'S ORIGINAL ENUMERATION SCHEMES 

Zeilberger [59] developed the notion of enumeration schemes and wrote the Maple package 
Wilf to automate their discovery. Roughly, enumeration schemes are a divide and conquer 
technique which aims to partition the class into smaller pieces from which recurrences can be 
derived. 

Take it £ Sk, suppose that n > k, and let 1 < i\ < i 2 < • ■ ■ < < n. In Zeilberger's original 
formalization of enumeration schemes, we divide Av n (B) into the sets 

A 7T (n;B;i 1 ,i 2 ,...,i k ) Av n (5) : p(l) = i^, . . . ,p(k) =V( fe )}. 

In words, A n (n; B; %\ % 12, ■ ■ ■ , ik) is the set of B-avoiding length n permutations that begin with 
the entries i±, 12, ■ ■ ■ ,ik, hi the order specified by n. For example, 

(1) ^312(9; B; 2, 3, 7) = {723x 4 x 5 x 6 x 7 x 8 x 9 E Av n {B)}. 

In order to make enumeration schemes more closely resemble generating trees and the in- 
sertion encoding, we consider a symmetry of his approach. Everywhere Zeilberger mentions a 
permutation we consider its inverse. Thus we should specify the set of restrictions, B, a set of 
small entries of some length, tt, and the positions in which the entries of tt occur. But instead 
of specifying the positions, we specify the gaps between the entries with a gap vector, g. After 
performing these transformations, our version of (1) is 

Z(B; 231; (1, 0, 3, 2)) = {x 1 23x 4 x 5 x (i lxsx 9 E Av 9 (B)}, 

and in general we are concerned with the sets 

Z(B; tt; g) = {p £ Av k+M (B) : p{gi + 1) = vr(l), . . . ,p( 9l + • • • + g k + k) = vr(fc)}, 

^A partially ordered set is said to be partially well-ordered if contains neither an infinite strictly decreasing 
subsequence (which is never possible for a permutation class) nor an infinite antichain. 
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where k is the length of ir and ||g|| denotes the sum of the components of g. Thus Z(B;7r;g) 
is the set of all i?-avoiding permutations of length k + ||g|| whose least k elements occur in the 
positions gi + 1, g\ + g 2 + 2, . . . , g\ + g 2 + • • • + 5% + k and form a 7r-subsequence. 

Not all pairs (7r,g) result in a nonempty Z-set. Following Zeilberger, for a length k permu- 
tation 7r we define 

J(tt) = {j £ [k + 1] : Z(B; vr; g) = for all g with gj > 0}. 

Thus Z(B;ir;g) is guaranteed to be empty if g does not "obey" ,y(vr), meaning that gj ^ 
for some j £ J (it). 

For example, consider the case B = {132}. Then 2 £ ^7(12) because if g 2 > then there is 
some entry between 1 and 2 in every permutation in Z(B;Tr;g), and this gives a 132-pattern. 
In order to check that ^7(12) = {2} we need merely observe that 312 and 123 avoid 132. Our 
approach in this example can easily be generalized to compute for any ir and B. 

Proposition 5.1. For any permutation ir and basis B, can be computed by inspecting 

the B-avoiding children of tt. 

Proof. Consider the vector h for which hi = for all i ^ j and hj = 1. If Z(B; tt; h) = then 
Z(B; vr; g) = for all g with gj > 0, so j £ J(tt). If instead Z(B; vr; h) + then j $ J(tt). □ 

For any r £ [A;], the set Z(B; n;(gi,..., gk+ij) embeds naturally (remove the entry 7r(r) and 
standardize) into 

(2) Z(S;st(vr - vr(r)); {g u .. . , g r _i, g r + g r+1 , g r+2 , . . . ,5fe+l)), 

where tt — 7r(r) denotes the word obtained from tt by omitting the entry 7r(r), so, for example, 
51342 — 1 = 5342. To make (2) easier to state, we define d r (ir) to be st(7r — 7r(r)) and let 

d r ((gi, ■ ■ ■ ,9k+l)) = (gi, ■ ■ ■ ,9r-l,9r + 9r+l,9r+2, ■ ■ ■ ,9k+l)- 

Sometimes the embedding of Z(B;ir;g) into Z(B;d r (ir);d r (g)) is a bijection. If this is true 
for all gap vectors g that obey ,y(vr), that is, that have gj = for all j £ <J(tt), then we say 
that 7r(r) is enumeration- scheme-reducible for tt with respect to B, or, for short, ES-reducible. 
(Zeilberger [59] refers to such entries as reversely deleteable.) We also say that a permutation 
with an ES-reducible entry is itself ES-reducible, and a permutation without an ES-reducible 
entry is ES-irreducible. 

For example, suppose again that B = {132} and consider the permutation 12. We have 
already observed that ^7(12) = {2}. Now we claim that the entry 1 is ES-reducible. The gap 
vectors that obey ^7(12) are those of the form (g%, 0, 33), and thus we would like to verify that 
the embedding of Z({132}; 12; (<?i,0, #3)) into Z({132}; 1; (51,53)) is a bijection. 

Take p £ Z({132}; 1; (gi , 53)) and consider inverting this embedding. In this case, that 
amounts to inserting the element 1 into position g\ + 1 and increasing all other entries of p 
by 1. Label the resulting permutation p' . For example, from g = (3,0, 1) and p = 52314 we 
obtain p' = 634125. 

We would like to show that p' avoids 132. To show this we consider all possible ways in 
which the new element 1 could participate in a 132-pattern. Clearly this entry must be the 
first entry in such a pattern. Now note that the 2 in p' cannot participate in this 132-pattern, 
because the 1 and 2 are adjacent. But then there is a 132-pattern in p' which uses the 2 instead 
of the 1, and thus p contains a 132-pattern, a contradiction. 

Thus we have shown that 

|Z n ,({132};12;( 9l , 52 , ff3 ))| = { | Zn _ l({ 132}; 1; ( 9l ,g 3 ))\ ^1 = 0. 

Although this example did not demonstrate it, detecting and verifying ES-reducibility by 
hand can be enormously tedious. Fortunately, it is also unnecessary. By adapting the approach 
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used in [54], we arrive at the following test for ES-reducibility that can be routinely checked 
by computer 10 . In it we let ||-B||oo denote the length of the longest permutation in B. 

Proposition 5.2. The entry 7r(r) of the permutation tt is ES-reducible if and only if 

|Z(J5;7r;g)| = \Z(B;d r {ir);d r ( g ))\ 

for all gap vectors g of the appropriate length that obey J(tt) and satisfy ||g|| < ||-B||oo — 1- 

Proof. If 7r(r) is ES-reducible then the claim follows by definition. To establish the other 
direction, suppose that 7r(r) is not ES-reducible, and choose g and p £ Z(B;d r (7r);d r (g)) so 
that g obeys J {it) but p cannot be obtained from a permutation in Z(B; tt; g) by removing 
7r(r) and standardizing. 

First form the (U-containing) permutation p' by incrementing each entry of p that is at least 
7r(r) by 1 and inserting 7r(r) into position gi + • • • + g r + r. Thus p' is the permutation that 
would have mapped to p, except that p' contains a pattern from B and thus does not lie in 
Z(B;tt; E ). 

Now pick some (3 £ B that is contained in p' , and choose a specific occurrence of (3 in p' . 
Note that since p = st(p' — vr(r)) avoids B, this occurrence of (3 must include the entry 7r(r). 
Let p" denote the standardization of the subsequence of p' formed by all entries that are either 
in the chosen occurrence of j3 or in tt (or in both), so p" contains a /5-pattern and lies in 
Z(0; 7r; h) for some ^(^-obeying h with ||h|| < ||-B||oo — 1- On the other hand, st{p" — n(r)) 
avoids B, which implies that \Z(B; d r (ir); h)| > \Z(B;ir;h)\, as desired. □ 



For example, consider the basis B = {132} again. In order to show that the 1 in it = 12 
is ES-reducible using this proposition, we first find that ^7(12) = {2} and then perform the 
following 10 computations: 



g 


|Z({132};12;g) 


|Z({132};l;di(g))| 


(0,0,0) 


1 


1 


(0,0,1) 


1 


1 


(1,0,0) 


1 


1 


(0,0,2) 


1 


1 


(1,0,1) 


2 


2 


(2,0,0) 


2 


2 


(0,0,3) 


1 


1 


(1,0,2) 


3 


3 


(2,0,1) 


5 


5 


(3,0,0) 


5 


5 



In a similar manner one can verify that the 1 in 21 is ES-reducible and that ,7(21) = 0. 
This gives the following enumeration scheme for Av(132): 



Zeilberger's approach in [59] used what he referred to as "logical reasoning," and while it is no less rigorous 
than this approach, Proposition 5.2 has the advantage of being very explicit. 
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|Z({132};0;(n))|, 
91-1 

£|Z({132};l;(i,(7i-i-l))|, 

i=0 
91-1 

\Z({132};21;(i, gi -i-l,g 2 ))\ 

i=0 

92-1 

+ J2 \Z{{132};l2;{ 9l ,i,g 2 -i-l))\, 

i=0 

9i 

£|Z({132};l;(i,«fc + g 2 - i - 

6. Extending enumeration schemes 

We will replace the sets in this section, giving us a more powerful version of enumer- 

ation schemes that can be found automatically with the Maple package WilfPlus. 

In order to motivate this change, we first consider a shortcoming of J {it). Let B = 
{1342, 1432}. It can be shown easily, even by hand, that 12 is ES-irreducible. To do so, first 
note that J {VI) = 0, as witnessed by the permutations 312, 132, and 123. Now consider the 
set Z{B; 12; (0, 2, 0)). This set is empty, but removing 1 gives the nonempty set Z{B; 1; (2, 0)) 
while removing 2 gives the nonempty set Z{B; 1; (0,2)). Indeed, this reasoning generalizes to 
show that all permutations of the form G m 12 are ES-irreducible, so Av(1342, 1432) does not 
have a finite enumeration scheme, at least in Zeilberger's original sense. 

Zeilberger's enumeration schemes fail in the previous example for a very simple reason: 
^7(12) is too coarse to capture the fact that (0, 2, 0) is not a valid gap vector for a U-avoiding 
descendant of 12. We remedy this problem with the following definition. 

Definition 6.1. The entry ir{r) of the length k permutation tt is said to be ES^ -reducible if 

|Z(B;7r;g)| = |Z(B;d P (7r);d r (g))| 

whenever Z n {B;ir;g) is nonempty. Further, we say that the permutation tt is ES^ -reducible if 
it contains an ES^ -reducible entry, and ES^ -irreducible otherwise. 

We then replace (for now) the set J by 

0(7r) = {g:Z(J3;7r;g)^0}. 

Recall that for ir{r) to be ES-reducible, it had to satisfy (3) for all g that contained 0's in the 
positions specified by J {n). Clearly if g € Q{ir) then g obeys J~{n), but there can be gap vectors 
that obey J{tt) and do not lie in Q{tt), as in our previous example with B = {1342,1432}. 
Thus we have obtained a weaker condition by requiring the satisfaction of (3) less often. 

The proof of Proposition 5.2 carries over to this context to give the analogous result on 
testing for ES + -reducibility. 

Proposition 6.2. The entry ir{r) of the permutation tt is ES^ -reducible if and only if 

\Z{B;7r;g)\ = \Z{B; rf r (vr); d r {g))\ 
for all g G G{it) with ||g|| < ||-B||oo — !• 



Sn(132) = 

|Z({132};0;( 51 ))| = 
|Z({132};1;( 51 ,<7 2 ))| = 
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One can view Q(tt) as an lower order ideal 11 of N' 71 "' -1-1 under the product order, where 
(xi, . . . , Xk) < (yi, ■ ■ ■ ,yk) if and only if xi < yi for all i G [k]. Therefore we carry our 
definitions about permutation classes over to this context. In particular, we say that the basis 
of G{it) is the set of minimal vectors not in £(7r), and if B is a set of vectors then we write 
Av(B) to denote the set {g : g ^ b for all b G B}. 

For example, let us compute the basis of £7(12) when B = {1342, 1432}. As already observed, 
(0, 2, 0) does not lie in £(12). This gap vector is minimal in N 3 \£(12) because Z(B; 12; (0, 1, 0)) 
is nonempty. To show that the basis of £(12) is precisely (0,2,0), it suffices to note that the 
permutation 

3 4 • •• ( 91 +2) 1 (si + 3) 2 ( 9l +4) ( 9l + 5) • • • (51 + g 2 + 3) 

avoids B, so (51,1,52) G £(12) for all 51,52 G N. Thus we have shown that £(12) = 
Av((0,2,0)). 

Now that we have computed £(12), it is not hard to check that 2 is ES + -reducible for 12. 
In order to do so we need to show that the embedding in question is a bijection for all gap 
vectors (51,52,53) with g 2 < 1. 

Suppose to the contrary that the embedding is not onto and take p G Z{B; 1; (51, 52 + 53)) 
that is not mapped to. In other words, the permutation p' obtained from p by inserting 2 into 
position 51 + 52 + 2 and incrementing all the entries of p of value at least 2 contains a 1342 or 
1432-pattern. Since p avoids {1342, 1432}, this pattern must involve the entry 2. First, the 2 
cannot play the role of the "2" in such a pattern, because then the 1 would be forced to play 
the role of the "1," and there can be at most one entry between the 1 and 2 since 52 < 1. 
The only other possible role for the 2 is as the "1," but in this case we could substitute the 1, 
thereby finding a U-pattern in p, a contradiction. 

Of course, we can replace by £(vr) only if we are able to work with £(vr). Because N' 71 "'"^ 1 
is partially well-ordered by the product ordering 12 , the basis of £(vr), which is by definition an 
antichain, must be finite. However, this basis may be quite large, or may contain vectors with 
large components. For example, another way to state the Erdos-Szekeres theorem [22] is that 

£(0) = Av(((j-l)(A;-l) + l)) 

when B = {12 • • • j, k ■ ■ ■ 21}. While this does not preclude effective computation of £(vr), it 
does suggest that such computations could be time consuming. 

It happens that we can circumvent this problem by replacing £(vr) by a different set of 
gap vectors. First we return to the way in which £(vr) is used. By our definitions, if n(r) is 
ES + -reducible for ir then 



\Z(B-v S )\-l ° ifg^W, 
|^,7r,gj|-| ^(^.^(^.^(g))! otherwise. 



This equality shows that when enumerating the S-avoiding descendants of ir with gap vector g, 
we first check to see if g lies in £(vr). If g ^ £(vr) then we can be sure that no such descendants 
exist. If g G £(vr), then Z(B;7r;g) is in one-to-one correspondence with Z{B\ d r (ir); d r (g,)). 
Note that if Z(B;ir;g) and Z(B; d r (ir); d r (g)) are both empty then they are trivially in one- 
to-one correspondence, so we could instead use the recurrence 

if gi G{7r) and dr(g) G G(d r (ir)), 



\7(n \\ - / if g^£(vr 

|^,7T,gj|-j \z(B;d r (Tr);d r (g))\ otherwise. 



This equality shows that instead of considering £(7r), we can look at the larger set of gap 
vectors for which either Z(B;n;g) 7^ or Z(B;d r (n);d r (g)) = 0. Unfortunately, this set 



11 This means that x € G(t) whenever x < y for some y £ Q(n). 

12 This fact, which is not difficult to prove, can be found many places, for example Nash-Williams [43]. 
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need not be an ideal 13 , so we consider the largest lower order ideal of N' 71 "^ 1 for which these 
conditions hold: 

g r {jT) = {g £ N W+1 : Z(B; n; h) + or Z(B; d r {ir)] d r (h)) = for all h < g}. 

Note that G(ir) C g r (ir): if g £ <?(tt) then Z(5;7r;g) / 0, so 7r; h) / for all h < g, so 

g £ Gr(ff)- With this observation we have 



\Z(B; 7r;g) 



ifg£^ r (vr), 
\Z(B; d r (ir); d r (g))| otherwise, 



if 7r(r) is ES + -reducible for tt and U. 

This new set has several advantages over G(ir). For one, it may be considerably smaller, 
thus simplifying the scheme. More importantly, the following bound on basis elements implies 
that G r {^) can be found automatically. 

Proposition 6.3. Suppose that ir(r) is an ES^ -reducible entry in tt (with respect to the per- 
mutation class with basis B). Then each basis vector b of Q r (it) satisfies ||b|| < ||-B||oo — 1- 

Proof. Suppose to the contrary that £/ r (7r) has a basis vector b with ||b|| > Hi?!^. Then 
Z(B; 7r; a) = and Z(B; d r (n); d r (&)) ^ for some a < b since b ^ G r (ir). However, because b 
is a basis vector for G T {ir), every a < b lies in G r (ir), so Z(B; tt; a) ^ or Z(B; d r (Tr);d r (&)) = 
for these vectors. If Z(B; d r (tt) ; d r (a) ) = for one of these vectors then Z(B; d r (ir); d r (h)) must 
also be empty, but this contradicts the fact that b ^ G r (^), so Z(B;7t;sl) ^ for all a < b. 
Therefore Z(B;n;b) = and Z(B; d r (ir); d r (h)) / 0. 

Now, since ir(r) is ES + -reducible and Z(B; it; a) ^ for all a < b, we know that there is a 
bijection between Z{B;tt]Sl) and Z(B;d r (Tr);d r (a.)) for all a < b. Thus we have the following 
diagram. 

Z(B;7r;b) = Z(B; d r (ir); d r (b)) ^ 



Z(B; tt; a) ^ < > Z(B; d r (7r); d r (a)) ^ 

The rest of the proof is similar to the proof of Proposition 5.2. Choose a permutation 
p £ Z(B;d r (-K);d r (h)) and form p' by incrementing each entry of p that is at least 7r(r) by 1 
and inserting 7r(r) into position g\ + • • • + g r + r. Choose a specific occurrence of some (5 £ -B 
in p'. Since p avoids 1?, this occurrence of (5 must involve the entry 7r(r). Let p" denote the 
standardization of the subsequence of p' given by the entries from tt together with the entries 
from the chosen occurrence of (3. Therefore p" contains a /3-pattern and lies in Z(0; tt; a) for 
some a < b with ||a|| < ||-B||oo — 1- However, this is a contradiction because d r (p") avoids 
B and thus lies in Z(B;d r (7r);d r (a)), and we have assumed that d r is a bijection between 
Z(B;vr;a) and Z(B;d r (ir);d r (a)). □ 

We conclude this section by writing out the enumeration scheme that we have derived for 
Av(1342, 1432). 



13 An example of this occurs with B — {231, k ■ ■ ■ 21}. In Proposition 8.1 we observe that the set of g 6 N 3 for 
which Z(B; 21; g) / is Av((0, 1, 0), (fe-2, 0, 0)) while the set of g € N 2 for which Z{B; 1; g) = is Av((fe-1, 0)), 
so the set of g £ N 3 for which either Z(B; 21; g) / or Z{B; 1; di(g)) = is Av((0, 1, 0)) \ {(k - 2, 0, 0)}. 



ENUMERATION SCHEMES FOR RESTRICTED PERMUTATIONS 



13 



|Z({1342,1432};0; (n))|, 
9i-l 

|Z({1342,1432};l;(z )5l -z-l))|, 

i=0 
91-1 

|Z({1342,1432};21;(i,(/i -i-l,g 2 ))\ 

i=0 

92-1 

+ Y |^({1342,1432}; 12; ( gi ,i,g 2 - % - 1))|, 

i=0 

91-1 

£ |Z({1342, 1432}; 1; (i, ffl + £ 2 - i - 1))| 
i=0 

+2|Z({1342,1432};1;( 51)52 -1))|. 

7. A COLLECTION OF FAILURES 

In this section we collect numerous negative results needed to justify the lack of inclusions 
in Figure 1. The class Av(123) shows immediately that classes can have finite enumeration 
schemes without having regular insertion encodings. An example of a class with a regular 
insertion encoding but without a finite enumeration scheme is given by Av(1234, 4231). It can 
be computed that this classes lies in 5,6(4), but the following proposition shows that it does 
not have a finite enumeration scheme. 

Proposition 7.1. For all k, the permutation k ■ ■ ■ 21 is ES^ -irreducible for Av(1234, 4231). 

Proof. Let ir = k ■ ■ ■ 21. First we show that the entries ir(r) = k — r + 1 for r G [k — 1] are 
ES + -irreducible. Let g denote the vector in N fc+1 which is identically zero except for g^-r+i 
and <7fc_ r+ 2, which are both 1 (these two components correspond to the gaps on either side of 
ir(k — r + 1)). Now observe that there are two permutations in Z({1234, 4231}; d r (ir); d r (g)): 

st(k(k — l)---(k — r + 2)(k + l){k + 2){k - r) ■ ■ ■ 21), and 
st(k(k - 1) • • • (k - r + 2){k + 2)(jfc + l){k - r) • • ■ 21), 

while Z({1234, 4231}; 7r; g) contains only one permutation, 

k(k - 1) • • • (k - r + 2)(k + l)(k - r + l)(k + 2)(k - r) ■ ■ ■ 21. 

Thus g G g r {it) but Z({1234,4231};vr;g) and Z({1234, 4231}; d r (vr); d r (g)) are not in one-to- 
one correspondence, so 7r(r) is not ES + -reducible for any r G [k — 1]. 

In order to show that ir(k) = 1 is not ES + -reducible, consider the gap vector g = (0, 0, . . . , 0, 3, 0). 
Then |Z({1234, 4231}; vr; g)| < |Z({1234, 4231}; 4 (tt); 4(g))|, finishing the proof. □ 

The substitution decomposition approach appears, at least on the surface, completely in- 
dependent from the other three methods. The class Av(321, 2341, 3412, 4123) has a finitely 
labeled generating tree by Theorem 2.1 (and thus it also has a finite enumeration scheme and 
a regular insertion encoding), but it contains infinitely many simple permutations 14 . 

Moreover, the separable permutations defined in Section 4, which contain only three simple 
permutations, possess neither a finite enumeration scheme nor a regular insertion encoding. 
To see that they do not possess a regular insertion encoding, we need only note that they have 
an algebraic generating function (so, for this purpose, Av(132) would work just as well). 



s n (1342, 1432) = 
|Z({1342,1432};0;(< 71 ))| = 

|Z({1342,1432};l;( ffl , 52 ))| = 



These simple permutations can be defined as the standardizations of even-length initial segments of the 
sequence 4, 1, 6, 3, 8, 5, . . . , 2k + 2, 2k — 1, . . . . 
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To establish that this class does not have a finite enumeration scheme, thereby completing 
our list of negative examples, we show that every permutation which consists of an increas- 
ing sequence followed by a decreasing sequence is ES + -irreducible for Av(2413, 3142). (This 
set forms a permutation class itself, with basis {213,312}.) Therefore, not only do the sep- 
arable permutations not have a finite enumeration scheme, but for each n there are 2 n_1 
ES + -irreducible permutations. Moreover, one can observe (either from the definition or the 
basis) that this class is invariant under the eight permutation class symmetries, so none of 
these offer any simplification. 

Proposition 7.2. Every vr G Av fe (213, 312) is ES+ -irreducible for Av(2413, 3142). 

Proof. Take ir G Avfc(213, 312). First we show that 7r(r) is not ES + -reducible for any 2 < 
r < k. If 7r(r — 1) > 7r(r), consider the gap vector g with all components except for 
g r = g r +i = 1- Then Z({2413, 3142}; ir; g) contains at most one permutation, because to 
avoid 2413, the smaller of these new entries must be to the left of the larger one. However, 
Z({2413, 3142}; d r (tt); ci r (g)) contains 2 permutations (the most possible). The case where 
7r(r — 1) < 7r(r) can be handled similarly with the gap vector that is except for g r ~i = 
g r = 1. Finally, 7r(l) is not ES + -reducible because Z({2413, 3142}; ir; (2, 1,0,..., 0)) contains 5 
permutations - every ordering of the new entries is allowed except 132, because that would give 
rise to a 2413-pattern - while Z({2413, 3142}; d\ (tt); (3, 0, ... , 0)) contains 6 permutations. □ 

8. AN ASSORTMENT OF ENUMERATION SCHEMES SUCCESSES 

Here we present several examples of finite enumeration schemes. In these presentations, we 
adopt the following pictorial representation. If ir is ES + -irreducible, so \Z(B; it; g)| is computed 
by summing over the £>-avoiding children of tt, then we draw a solid arrow from tt to each of 
its children. If the entry 7r(r) if ES + -reducible in tt then we draw a dashed arrow from tt to 
d r (7r), label this arrow with d r , and indicate the basis of G r {ir) beneath tt. For example, the 
enumeration scheme for Av(1342, 1432) is shown in Figure 3. We also define the depth of an 
enumeration scheme to be the least integer k so that every permutation of length at least k is 
ES + -reducible. 






g 2 (0,2,0) 

Figure 3. The enumeration scheme for Av(1342, 1432) 



i 



FIGURE 4. The enumeration scheme for the set of all permutations, Av(0) 
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v 

di 
i 

/ 

1 

g t (a,b) 

Figure 5. The enumeration scheme for Av(M a ^) 






i / \ \ \\ x / 

3124 4312 3142 3412 

g 2 (0,0,0,0,1) 



Figure 6. The enumeration scheme for Av(1234) 

As can be seen in Figure 3, we include the empty permutation, 0, in our diagrams. Although 
this rarely has no more effect than making our diagrams consume more vertically space, there 
are a few exceptions. One is the set of all permutations, Av(0), which has the enumeration 
shown in Figure 4. Another exception is Av(M Qj ft) where M a ^ = {(3 £ 5 a _|_&_|_i : (3(a + 1) = 1}. 
This class was first counted by Mansour [35]. Its enumeration scheme is shown in Figure 5. 

The scheme for Av(1234), which generates sequence A005802 in the OEIS [47], is shown in 
Figure 6. 

The 321, hexagon-avoiding permutations, which can be defined as 

Av(321, 46718235, 46781235, 56718234, 56781234), 

were first introduced by Billey and Warrington [10], who showed how to compute the Kazhdan- 
Lusztig polynomials for them. Stankova and West [48] proved that the number, s n , of these 
permutations of length n satisfies 

s n = 6s n _i - lls n _ 2 + 9s n _ 3 - 4s n „ 4 - 4s n ,_5 + s n _ 6 

for all n > 7, which gives sequence A058094 in the OEIS [47] . (So this complicated scheme pro- 
duces only a sequence with a rational generating function.) Later, Mansour and Stankova [37] 
counted 321, 2/c-gon-avoiding permutations for all k. 
The freely braided permutations are the class 



Av(3421, 4231, 4312, 4321). 
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132 312 

g t (1,0, 0,0) 
/I 



I 

1234 1243 1423 * _ 

g t (3, 1,0,0,0), 

(4,0,0,0,0) dl _ 




41235 41253 41523 45123 

g t (1,0,0,0,0,0), 

(0,2,1,0,0,0), 

(0,3,0,0,0,0) 



(1,0,0) 



Figure 7. The enumeration scheme for the 321, hexagon-avoiding permutations 




g 2 (1,0,0,0) g £ (2, 0,0,0) g 2 (1,0,0,0) g 2 (1,0,0,0), 

(0,1,0,0) 

Figure 8. The enumeration scheme for the freely braided permutations, 
Av(3421, 4231, 4312, 4321) 

This class was introduced by Green and Losonczy [25] and also arises in the work of Tenner [53]. 
Mansour [36] found that the freely braided permutations have the generating function 

1 - 3x - 2x 2 + (1 + x)y/l-4x 

1 - Ax - x 2 + (1 - x 2 )y/l - 4x ' 

(Sequence A108600 in the OEIS [47].) They also have an enumeration scheme of depth 3, 
shown in Figure 8. 
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g t (0,0, 1,0), g ¥ (0,0,1,0), 
(0, ft -3, 0,0) (0,1,0,0) 

(c) Av(231, lft • • • 32) 



Figure 9. Enumeration schemes for three Wilf-equivalent classes 

Chow and West [15] showed using generating trees that the classes in Figure 9 are Wilf- 
equivalent, that their generating functions are rational 15 , and that these generating functions 
can be expressed as quotients of Chebyshev polynomials. Krattenthaler [30] gave another 
proof via a bijection to Dyck paths (in fact, he found the bivariate generating functions for 
132-avoiding permutations by length and number of copies of 12 • • • k and for 123-avoiding 
permutations by length and number of copies of (k — 1) • • • 21k). Around the same time as that 
work, several other authors studied these and similar classes: Jani and Rieper [26], Mansour 
and Vainshtein [38, 39], and Robertson, Wilf, and Zeilberger [44]. Deutsch, Hildebrand, and 
Wilf [18] used these results in their study of the longest increasing subsequence problem for 
132-avoiding permutations. 

For any fixed k, WilfPlus can automatically and rigorously derive enumeration schemes for 
these classes. This makes it easy to make conjectures for the general form of these enumeration 

15 This fact can be verified quite quickly: by inverting the permutations in Av(321, 23 ■ ■ ■ fcl) one obtains 
the class Av(321, fcl2 • • • (ft — 1)), which satisfies the hypotheses of Theorem 2.1, and thus also of Theorem 3.1. 
Note, however, that while symmetries of these classes always have finitely labeled generating trees, and thus 
also regular insertion encodings, the complexity of their generating trees and insertion encodings increases with 
ft, while the complexity of their enumeration schemes stay fixed. 
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schemes, although they must be verified by hand. We carry this out for one of these classes 
below. 

Proposition 8.1. The enumeration scheme for Av(231, k ■ ■ ■ 21) is as shown in Figure 9 (b). 

Proof. Let B = {231, k ■ ■ ■ 21}. First we verify the computations of G r (ir) given in the diagram. 
For 12, the diagram shows that Qi(12) = N 3 . In any permutation in Z(B; 12; (gi, g 2 , 53)), the 
entries before the 2 must be in decreasing order from left to right as any ascent before the 2 
would give rise to a 231-pattern. Then, since these permutations must avoid k ■ ■ ■ 21, there can 
be at most k — 2 new entries before the 2, so {gi,g 2 ,gs) ^ ^(12) whenever g\ + g 2 > k — 1. 
The i?-avoiding permutations 

91 + 92 + 2, • • • , gi + 4, g 2 + 3, 1, g 2 + 2, . . . , 4, 3, 2, g\ + g 2 + 3, g\ + g 2 + 4, . . . , gi + g 2 + gz + 2 

then show that £?(12) is Av({(gi, g 2 , 93) '■ gi + g 2 > k — 1}). It can be shown similarly that 
C/(l) = Av((/c — 1, 0)). This shows that <7i(12) = N 3 because Z(B; 1; (ii(g)) is empty whenever 
Z(B; 12; g) is empty. 

The computation for 21 is similar. One first checks that 

0(21) = Av((0,l,0),(fc -2,0,0)). 

Since G(i) = Av((/c — 1,0)), the set of gap vectors g for which either Z(i?;21;g) 7^ or 
Z(B; 1; di(g)) = is Av((0, 1,0)) \ {(k - 2, 0, 0)}. This implies that C?i(21) = Av((0, 1,0), (k - 
2,0,0)). Now we must verify the ES + -reducible entries. The 12 case is clear because the only 
way 1 can participate in a 231 or k ■ ■ ■ 21-pattern is as the minimal entry, and in either case 
the 2 could play the same role. For 21, first note that the 2 also cannot participate in any 231 
or k ■ ■ - 21-pattern if g S <5i(21): it cannot be the "2" in a 231 because there can be nothing 
between the 2 and the 1, and it cannot be the "2" in a k ■ ■ ■ 21-pattern because our restrictions 
on g prevent there from being enough entries to the left of the 2 to accommodate such a 
pattern. This shows that the 2 can only possibly be the minimal entry in a forbidden pattern, 
but then the 1 could play the same role. □ 

In addition to the examples already mentioned, there are several other interesting classes 
avoiding a pair of permutations of length four 16 : 

(1) Kremer and Shiu [32] proved that there are four classes of this form counted by (4 n ~ x + 
2)/3. These all have finite enumeration schemes: Av(1234, 2143) and Av(1432, 2341) 
have schemes of depth 3, Av(2341, 4321) has a scheme of depth 4, and Av(2143, 4123) 
has a scheme of depth 6. 

(2) Kremer [31] (see also Stanley [52, Exercise 6.39.1]) completed the characterization of 
the classes defined by avoiding two permutations of length four that are counted by the 
large Schroder numbers. Up to symmetry, there are ten such classes, seven of which 
have finite enumeration schemes: 



'The characterization of the Wilf-equivalences between these classes has recently been completed by Le [33]. 
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Class 



Finite enumeration scheme? 



Av(1342, 2341) 
Av(1342, 1432) 
Av(2341, 2413) 
Av(2413, 3142) 



Yes, of depth 3 

Yes, of depth 2, shown in Figure 3 
No 



No, these are the separable permuta- 
tions considered in Proposition 7.2 
No 

Yes, of depth 4 

Yes, of depth 2 (Knuth [29] proved 
that these are precisely the permuta- 
tions that can be sorted by an input- 
restricted deque) 
Yes, of depth 3 
Yes, of depth 2 
Yes, of depth 4 



Av(2431, 3241) 
Av(3241, 3421) 
Av(3241, 4231) 



Av(3412, 3421) 
Av(3421, 4321) 
Av(3421, 4231) 



(3) A permutation is said to be skew-merged if it is the union of an increasing subsequence 
and a decreasing subsequence. Stankova [49] was the first to prove that the skew- 
merged permutations are given by Av(2143, 3412). Later Kezdy, Snevily, and Wang [27] 
gave another proof of this result using Foldes and Hammer's characterization of split 
graphs [23]. Atkinson [5] showed that the generating function for this class is 



(1 - 2x)Vl -4x' 

(sequence A029759 in the OEIS [47]). This class has an enumeration scheme of depth 
four. 



We have developed an extension of Zeilberger's enumeration schemes and provided the mech- 
anisms for their automatic generation, a task implemented in the accompanying Maple package 
WilfPlus, which is available at http://math.rutgers.edu/~vatter/. As the examples in 
Section 7 demonstrate, there remain considerable differences in the applicability of enumera- 
tion schemes, substitution decompositions, and the insertion encoding. This obviously suggests 
the following question. 

Question 9.1. Is there a systematic method of permutation class enumeration which is appli- 
cable to all classes with finite enumeration schemes, all classes with regular insertion encodings, 
and all classes with only finitely many simple permutations? 

Additionally, one would like such a method to be invariant under the eight permutation 
class symmetries, but Question 9.1 is probably demanding enough in the form above. 
We conclude by collecting three of the questions raised earlier. 

Question 9.2. Is every sequence produced by a finite enumeration scheme holonomic? 

As demonstrated by the examples in Section 8, another interesting question is the equivalence 
problem: 

Question 9.3. Is it decidable whether two finite enumeration schemes produce the same se- 
quence? 

Brlek, Duchi, Pergola, and Rinaldi [14] consider the equivalence problem for generating trees 
with infinitely many labels. 

Although only tangentially related to the main thrust of this article, the following question 
nevertheless seems intriguing enough to warrant its inclusion. 

Question 9.4. Is it decidable whether a class contains only finitely many simple permutations? 



l-3x 



9. Conclusion 
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